In this exploratory study, WhatsApp survey data is compared to previously collected face-to-face, telephone and Web survey data. We did not find significant effects for non-differentiation and acquiescence between WhatsApp mode and the other modes. For acquiescence, respondents seem to show least bias in the WhatsApp survey and most bias in face-to-face interviews. For potential social desirable behavior, our findings are mixed. As regards to user experience, respondents expressed positive attitudes towards convenience of answering questions, but open-ended questions that need much text were not appreciated, and some respondents were worried about their privacy.